Group-coherent memory

ABSTRACT

Operating a data distribution including a data distribution module and a plurality of host-bus adapters coupled to the data distribution module can include defining a coherent group that includes a set of members that includes the plurality of host-bus adapters; providing a group-coherent memory area in each of the set of members; and initiating a one-to-all broadcast message from a one of the plurality of host-bus adapters to all of the set of members when the one of the plurality of host-bus adapters requests a write to its local group-coherent memory area. The group-coherent memory area in each of the set of members is physically mirrored with a temporal coherence and no semaphores or access enables are required to achieve the temporal coherence of the coherent group.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of U.S. application Ser. No.15/175,685, filed Jun. 7, 2016. This application is acontinuation-in-part of U.S. application Ser. No. 15/262,391 filed Sep.12, 2016, which claims priority benefit from U.S. ProvisionalApplication No. 62/216,999, filed on Sep. 10, 2015, and U.S. ProvisionalApplication No. 62/217,001, filed on Sep. 10, 2015, and U.S. ProvisionalApplication No. 62/217,003, filed on Sep. 10, 2015, and U.S. ProvisionalApplication No. 62/217,004, filed on Sep. 10, 2015 and U.S. ProvisionalApplication No. 62/241,112, filed on Oct. 13, 2015. The patentapplications identified above are incorporated here by reference in itsentirety to provide continuity of disclosure.

BACKGROUND OF THE INVENTION

A particular memory model is advantageous in computing architectures formultiple tasks, whose threads are distributed across multiple andseparate hosts, requiring access to the same data. Such tasks may beviewed as a working group, dealing with different aspects of the sameproblem while reading from and writing to the same relative memorylocations within each individual host. In a cluster-wide sharedarchitecture, this expanded requirement is often handled by specializedhardware and software in addition to the usual networking hardware forinterconnecting the cluster. The goal is to move data between differenthosts across the entire cluster such that a portion of local memory ineach host is maintained as a “mirror image” of the same relative memoryin any other host.

Moving memory contents around to meet coherency needs can materiallyslow a parallel application. What is desired is a method ofautomatically updating mirrored copies across a computing cluster withwithout the addition of specialized hardware.

SUMMARY OF THE INVENTION

There is a need for the following embodiments of the present disclosure.Of course, the present disclosure is not limited to these embodiments.

According to an embodiment of the present disclosure, a method comprisesoperating a data distribution system including a data distributionmodule and a plurality of host-bus adapters coupled to the datadistribution module including defining a coherent group that includes aset of members that includes the plurality of host-bus adapters;providing a group-coherent memory area in each of the set of members;and initiating a one-to-all broadcast message from a one of theplurality of host-bus adapters to all of the set of members when the oneof the plurality of host-bus adapters requests a write to its localgroup-coherent memory area. According to another embodiment of thepresent disclosure, an apparatus comprises a data distribution systemincluding a data distribution module and a plurality of host-busadapters coupled to the data distribution module, wherein operating thedata distribution system includes defining a coherent group thatincludes a set of members that includes the plurality of host-busadapters; providing a group-coherent memory area in each of the set ofmembers; and initiating a one-to-all broadcast message from a one of theplurality of host-bus adapters to each of the set of members ensuringthat when the one of the plurality of host-bus adapters request a write,to update its local group-coherent memory area, the one-to-all broadcastmessage maintains temporal memory coherency across all of the set ofmembers of the coherent group.

These, and other, embodiments of the present disclosure will be betterappreciated and understood when considered in conjunction with thefollowing description and the accompanying drawings. It should beunderstood, however, that the following description, while indicatingvarious embodiments of the present disclosure and numerous specificdetails thereof, is given for the purpose of illustration and does notimply limitation. Many substitutions, modifications, additions and/orrearrangements may be made within the scope of embodiments of thepresent disclosure, and embodiments of the present disclosure includeall such substitutions, modifications, additions and/or rearrangements.

BRIEF DESCRIPTION OF THE DRAWINGS

In the detailed description of the preferred embodiments presentedbelow, reference is made to the accompanying drawings.

FIG. 1 illustrates the data-distribution system (DDS) tight-clusterinterconnect.

FIG. 2 depicts a coherent memory update process.

FIG. 3 shows the several components and their interconnections thatdefine a data distribution module (DDM).

FIG. 4 lays out internals of the data distribution module's (DDM's)input stage.

FIG. 5 illustrates the mechanism from the FIFOs (First In First Outbuffers) in the interior of the data distribution module (DDM) to theoutput stage.

FIG. 6 depicts an output process that allows priority interrupts.

FIG. 7 defines the control-frame protocol fundamental to the operationand control of the distribution of messages.

FIG. 8 gives an example of a type of fabric of data distribution modules(DDMs).

DETAILED DESCRIPTION OF THE INVENTION

In the description that follows, like parts are marked throughout thespecification and figures with the same numerals, respectively. Thefigures are not necessarily drawn to scale and may be shown inexaggerated or generalized form in the interest of clarity andconciseness.

In FIG. 1, The invention relates generally to the field of methods anddevices for maintaining coherence between mirrored copies of a task's orprogram's memory within a cluster of interconnected computers or hostdevices.

The invention presented in this disclosure provides a coherent groupmemory implemented specifically for a tight cluster of servers or otherhost devices such as found in database installations, high-performancecomputing applications, and anywhere parallel or cooperative programmingmay be needed. The coherency mechanism described in this disclosure issupported by the same hardware used to interconnect the cluster and isimplemented by defining a coherent memory area in each host and ensuringthat a write to that area by any thread in any host initiates aone-to-all broadcast message from the host requesting the write to allhosts in the cluster, including itself. That is, the method is based ona single group whose task it is to maintain temporal memory coherencyacross the cluster.

The interconnect mechanism describes a message-distribution system orDDS consisting of host bus adapters (HBAs) and a data-distributionmodule (DDM) where the HBAs provide an interface mechanism between ahost and the DDM. The concept, which may also be called “virtual sharedmemory”, is based on the unique multicast mechanism inherent in the DDS,which can include a data-distribution module (DDM) with host-busadapters (HBAs) mediating between hosts and the DDM.

The following sections describe how to achieve a coherent memorymirrored across multiple stations when defined by a single group. Thephysical memory in the DDS is mirrored (each host has a copy of thecoherent memory) where each host in the system belongs to the singlecoherence group that, while physically mirrored across all hosts orservers, is accessed as if it were a single memory asset shared by theentire cluster. The difference is that no semaphores or access enablesare required to achieve a temporal coherence of the cluster's “shared”memory.

At the architectural level, the DDS is based on multicast; that is,multicast is the native operational mode of the DDS in that everymessage injected into the system is treated as multicast having one ormore destinations. This native multicast mechanism is built in at thehardware level or physical layer of the DDS. Messages are guided throughthe DDM (from HBA to HBA) by means of a destination field in thestart-of-message (SOM) header, which directs the message along internalpaths from the input port to the specified output ports.

In addition to the multicast, multiple priority levels are built in atthe architectural design stage. The lowest and highest priorities arereserved for system functions such as flow control and systemmaintenance. The message priorities are maintained as separateinformation channels from a transmitting HBA through the DDM to thereceiving HBA.

A third design feature of the MDA is its support for “fast-prioritymessages” or FPMs which are single-frame messages whose purpose is tomaintain efficient control over system functions. In the output stage ofthe DDM and HBA, these messages are assigned priority P0 which meansthat, if queued in a P0 queue, they have transmission priority (eitherto the HBA or to the DDM).

Any message sent by a host that is meant to update the coherent memoryis assigned to channel P1 while any other messages involvingmemory-to-memory are assigned priority channels such as a P2 or P3channels with lower priority than channel P1. All flow control and othercritical semaphores take place on the P0 channel. Such control framesbypass the internal distribution mechanism of the DDM and carry outtheir specific functions without interfering with normal messagetraffic, other than introducing a one-frame delay. These features arediscussed in detail in the above-referenced patent application.

Based on the above description, a set of mirrored memory locationsdistributed over the cluster is maintained in a coherent fashion byensuring that any host updating its local copy of the coherent memoryalso sends out a one-to-all broadcast to all other hosts in the cluster.

The memory mechanism proposed here is meant to ensure a read-coherenceacross mirrored copies of a single, privileged group and the methodproperly supports coherence only in a tight computing cluster. There areno read requests to the local copy other that the usual reads supportedby the kernel. Additionally, there are no special write locks beyondnormal memory DMA writes.

The tight-cluster constraint may be relaxed by a simple handshakemechanism between the HBA and the host receiving a memory update and thesingle group may be expanded to multiple groups while retaining thesingle coherence group for maintaining coherent memory for a singlecooperative-computing task. However, these enhancements requirearchitectural changes to the interconnect and must be supported bysoftware enhancements to the application programming interface.

In summary, the memory model disclosed here maintains a group-based,system-wide coherent memory for those hosts having membership in thespecial coherent group. Coherence in this special group is achieved atthe architectural level of the DDM by operationally restricting the P1channel to accept only group-memory updates in the form one-to-allmessages. These updates are effectively broadcast from the host issuingthe update to all other hosts in the system. Due to the prioritymechanism, any memory update is sent from the updating HBA to the DDMwhere it is distributed to the remaining hosts without interference ordelays from any other system messages. A feature inherent in the way theDDM processes multicast ensures that the receiving HBAs willsimultaneously receive the same update information as long as they arerestricted to the P1 channel. Each HBA will pass on its copy of theupdate at the correct priority to the host by means of a DMA action,ensuring that the memory images across the cluster will be updated atthe same time.

There is no need for locks and semaphores in this basic model. Thus,coherency is maintained across all system hosts without effort beyondreserving the highest message priority for memory updates. The otherpriority channels are free to handle whatever message traffic isrequired and do not interfere with the coherency of the memory update.Thus, other memory locations may receive messages on the lower-prioritychannels, but such are not guaranteed to be coherent across theirrespective groups.

Otherwise stated, the basic concept supports multiple groups allowingother modes of communication between hosts (such as maintenance andreporting functions). The coherence group is a preferred group whoseonly access is by means of the P1 priority channel and this channel maybe reserved only for coherent memory updates. The memory images of theother groups are consistent in that a group write updates all imageswithin a group just as done for the coherent group. However, theseupdates are not guaranteed to maintain a strict read coherency as doesthe privileged group by virtue of its temporal coherency.

Referring to FIG. 1, DDS tight-cluster 150 consists of centraldata-distribution module (DDM) 100 connected to host-bus adapters (HBAs)120 via fiber connections 110. Each HBA 120 connects via connections 130to host or CSNE 140 (computing, storage, or network element) such thatthese elements are fully connected to each other by means of DDM 100.Each host or CSNE 140 contains an identical region of memory, perhapswith different absolute addresses within each host. It is thiscollection of memory images that is required to remain coherent withprecisely the same contents at any instant in time. There are severalclasses of “coherent” memory. The most common one may be termed “readcoherent” in that any time any host is allowed to read the same relativememory location, the same value is obtained; enforcing read coherenceusually requires the use of locks and semaphores. The most stringenttype of coherent memory is temporally coherent in that reads are notdelayed or controlled by semaphores, yet a random read by any host toits local copy of the coherent memory is guaranteed to return the samevalue.

Referring to FIG. 2 and assume for now that there is no other traffic onpriority channel P1 through DDM 220. Then a memory-update messagebroadcast from Host j 200 will enter HBA j 210 via connection 205 and beprepared according to the prescription disclosed. Each host receivingthe update is a member of a “coherent group” that is defined as aworking or cooperative group of processes (or threads or tasks), oneresiding in each host belonging to the group, such that groupcommunication is restricted to and reserved for priority channel P1 asdescribed in the above-referenced patent application.

Since the group-coherent update is a broadcast message wherein thetransmitting host is numbered among the recipients, the message isprepared with a group index referring to the subscription table entrywhere all bits are set, indicating that all exits in the DDM are totransmit copies of the update. This message then enters DDM viaconnection 215 with the start-of-message header (SOM) prepared with theaforementioned group index as its destination and a priority indicatingthe P1 priority channel and the offset into the mirrored copiesindicating the location in each mirrored image that is to receive theupdate. The end-of-message (EOM) is prepared as normal with a prioritydesignation of P1, a source index of j, and a CRC-32 computed over thedata portion of the update as described in detail in theabove-referenced patent disclosure.

The DDM behaves as described in the referenced disclosure and thecoherent update is distributed to each P1 output FIFO simultaneously asthere are no prior messages causing delays in any of the paths. Copiesof the update then leave all exit ports along connections 225 to eachconnected HBA such as HBA m 230, arriving in all HBAs, including HBA j210, simultaneously since there is no other traffic in the cluster todifferentially delay any of the updates. In a similar manner, since HBAm 230 and HBA j 210 are free to process any messages from the DDMwithout delay and such messages are written by direct-memory access(DMA) directly into the targeted memory areas at the uniformly specifiedoffset positions from the base address in each target host, such as hostm 240 via connection 235 and host j 200 via connection 205, the updatesarrive simultaneously in all copies of the group memory (within a smalldifferential jitter of a few nanoseconds due to inhomogeneities in thevarious physical paths).

The mechanism for the coherent update occurs when a process running ahost completes a calculation or receives a message (not directed to itsgroup-coherent memory) that is to be written out to the coherent group.The write in this case may be trapped by the operating-system kernel inone possible implementation of the process. The kernel then initiates abroadcast to all members of the coherent group, including a send toself. This update then undergoes the process described above so that allcopies of the group-coherent memory are updated synchronously or nearlyso within the time required for a single update to traverse the cluster.

The process disclosed herein does not prevent the use of semaphoresbased on the fast-priority message for maintaining flow control or othersystem functions described in the above-referenced patent disclosure.The process allows multiple simultaneous coherent updates from differenthosts. Such updates will, perhaps, reach the DDM simultaneously where anarbitration mechanism, disclosed in the above-referenced patentapplication, will maintain message order such that each message arrivesintact in a serial fashion. Precedence of simultaneous coherence updatesto same mirrored location must be resolved at the application level by,perhaps, including a priority of some sort within the body of themessage. Such issues do not alter the method or effectiveness of thedisclosed process.

Embodiments of this disclosure can include a method of maintaining acoherent memory for an interconnect system having the capability to (1)define a coherent group and allocate a corresponding mirror memory areain each member of the group; (2) broadcast messages such a group suchthat they arrive at their destinations simultaneously or nearly so; and(3) such that the group member initiating the message also sends thesame message to itself through the same mechanism. Embodiments of thisdisclosure can include a coherent memory update that takes place overthe same DDS that is used to interconnect the tight cluster. Embodimentsof this disclosure can include a method of coherent updates that arecarried out by means of a special coherent group. Embodiments of thisdisclosure can include a method of coherent memory update across a tightcluster that is lock- and semaphore-free, wherein updates take placeover a high-priority channel exclusively reserved for such updates.Embodiments of this disclosure can include a method of coherent updatesthat are initiated by a write request to a local copy of the coherentmemory. Embodiments of this disclosure can include a coherent memoryupdate that can be used in conjunction with existing tight-clusterinterconnects as an additional, add-on system requiring another set ofHBAs, connections, and the DDM adjacent to and parallel with an existingswitched interconnect.

FIG. 3 illustrates the data flow to, through, and from the DDM. Outputlines in the HBA are connected to the DDM's input stage, input box 310via fiber connections 300. Input box 310 extracts and decodes thedestination information from the SOM to select the partition(s) andpriority channels for queuing the incoming message in middle box 330.The message is then sent along connections 320 to chosen partitionmanagers 340 in middle box 330, where it is either queued, if a delay isrequired, or it is immediately passed on to output manager 370 in outputbox 360 via connections 350. The partition concept extends from middlebox 330 through output box 360. The priority structure is maintainedthroughout these partitions and priority interrupts, where ahigher-priority message from an HBA can interrupt a lower-priority oneby a simple path change made in input box 310, allowing the stream ofincoming frames on connection 300 to switch to the higher-priority queuewithin a partition manager 340.

Partition manager 340 makes a connection to output manager 370 in thesame partition when the SOM is cleared for transmission to output box360. Output manager 370 then queues the message frames in a FIFOcorresponding to the priority and exit port specified by the SOM'sdestination code. The message is then passed directly to the output portin output manager 370 where it is converted to a serial optical signaland sent along output fiber 380, or it is held in output manager 370should the port be busy with another transmission.

In greater detail, suppose HBA j sends a message prefixed by a SOMcontaining the destination, priority, and an offset. This message willappear on input line 400 and enter input box 310 on port 410 where it isconverted to an electrical signal, deserialized, and sent to inputprocessor channel 430 via parallel data lines 420. Input processorchannel 430 extracts the destination and priority from the correspondingfields in the SOM. The destination is an index into a subscription tablemaintained in input box 310 by the aforementioned maintenance messages.Input processor channel 430 retrieves the exit code from thesubscription table and decodes this information to obtain (1) the listof partitions that are to receive copies of the message and (2) the exitindices in the form of an exit map for each of the receiving partitions.

The exit map for each partition is sent to distributor 450 viaconnections 440 which contains a separate path for each of thepartitions. Distributor 450 sends a copy of the SOM accompanied by therelevant exit map to specified partition FIFO 470 via connection 460 andsets a mux in distributor 450 to connect the message channel to theindicated partition FIFO 470.

A consequence of the selection process effected by input processorchannel 430 and implemented by distributor 450 is to send the incomingmessage frames from channel j onto selected connection 460 to be queuedinto selected partition FIFOs 470 according to exits decoded from thedestination and priority fields carried by the SOM. Note that a SOM mayspecify a single priority and multiple partitions. All sub partitionsk,p belonging to input index j indicated in FIG. 4 will receive a copyof the message so directed.

Connection 432 transfers any FCM frame transmitted by HBA j directly tooutput box 360, bypassing middle box 330 to maintain proper flow controlfrom exit port j. Similarly, connection 434 transmits any responserequested by a maintenance message received by the input stage from HBAj for queuing in a maintenance output FIFO (see below).

Considering connections 320 in FIG. 3 together with connections 460 inFIG. 4 that are available to any input port, it becomes clear that amessage on connections 300 of FIG. 3 can be sent along a path to any ofseveral partition FIFOs 470 that are specified by the destination andpriority fields.

FIG. 5 illustrates the data flow from middle box 330 to output box 360as shown in FIG. 3 and subsumed in partition manager 340. PartitionFIFO[k,p,j] 510 where k specifies the partition, p the priority, and jthe input port, receives frames introduced by a SOM on input line 500 asdetermined by input processor channel 430. Each partition FIFO[k,p,j]510 has an associated queue processor 520 (indexed by the same {k,p,j}as the FIFO) that has access to the head or output frame residing inthat FIFO via connection 512. Queue processor 520 may then determine ifa frame is present or not, if the frame is a SOM or EOM, or simply adata frame.

When a SOM is identified as presenting a message with a set of exits {m}(for the sub partition k,p), this information is passed to correspondingarbiter[k,p] 540 via connections 524. Dotted line 550 representconnections to and from the remaining queue processors [k,p,{j}] where{j} represents a collection of indices representing all input ports inthe DDM.

Arbiter[k,p] 540 compares the requested set of exits {m} with aninternal exit map that maintains the state of the set of muxes 530corresponding to the entire set of Output FIFOs. If the bits in this mapcorresponding to the set {m} are not set, a release is granted bysending notification back to queue processor 520, which sets state ofmuxes 530 via lines 522 so that the SOM may be transferred to the set ofoutput FIFOs[k,p,{m}] 560 via selected connections 535. The set oflocations corresponding to {m} are then set in the map to preventconflicting messages from being released into output FIFO[k,p,j] 560.

If the set {m} of locations in the bit map are not all clear, therequest is not granted and the SOM must wait in partition FIFO[k,p,j]510 until such time that any conflicting messages have completed theirtransit from middle box 330 to output box 360.

The state machine in each queue processor 520 periodically queriesarbiter[k,p] 540 as long as a SOM remains at the front of the partitionFIFO[k,p,j] 510, ensuring that any pending message will be released assoon as the map bits {m} are clear.

Once the SOM has been released and transferred to specified output FIFOs560, the rest of the message follows by a handshake process controlled,for example, by read- and write-enable flags in the transmitting andreceiving FIFOs. These flags reflect the current state of theirrespective FIFOs in the preferred embodiment, with the read enable setwhen there is a frame present in partition FIFO[k,p,j] 510 and thewrite-enable set in output FIFO[k,p,m] 560 as long as there is room inthat FIFO for the next frame to be transferred. Note that all outputFIFO[k,p,m] 560 in the released set {m} must be write enabled fortransfer to take place.

The transfer continues, frame-by-frame, until an EOM is transmitted.This event, detected by the Queue Processor 520, clears the bits {m} inthe exit-port map in arbiter[k,p] 540, thus allowing any followingmessage access to those output FIFO[k,p,m] 560 that received the EOM.

FIG. 6 illustrates the role of output manager 370 in output box 360 as aprocess managing message traffic from all output FIFOs [k, {p}, m] 560to output port[k,m] 660 in partition k for the specified output m inthat partition. {p} is the set of message priorities supported by theMDS. This manager for partition k serves only exit m by mediating thetraffic from all priorities from 0 to P+1, where P is the number ofmessage priorities supported by the MDS. There is one such manager foreach of the n output ports in the MDS. FIG. 6 depicts the preferredembodiment with P=3, resulting in a total of 5 output FIFOs 630 in eachof the K partitions.

Inputs 600 from the set of Middle Box Partition Managers as shown inFIG. 5 send message frames to Output FIFOs 610 from each messagepriority as mediated by arbiters[k,p] 540 in partition k for any or allof the P priorities. Input 614 allows any maintenance message preparedby input processor channel 430 on input channel j to be queued in OutputFIFO [k,4,m] via connection 434 where m=j mod M so the response is sentto the requesting HBA j. Likewise, input 612 contains informationregarding any of the partition FIFOs 470 that generate a flow controlexception. Input 612 also transmits any FCM sent by an HBA and processedin input processor channel j 430 via connection 432. Both of thesenotifications are processed by flow control processor 620 which eitherformats an FCM for delivery to HBA j in the case of a notification frompartition FIFO 470 or controls the flow of frames to the outputport[k,m] in the case of a notification from the input processor channelj.

If one or more of the output FIFOs[k, {p}, m] contain at least oneframe, output selector[k,m] 640 selects appropriate output FIFO 610 or630 depending upon priority, via connections 635 for transmission tooutput port [k,m] 660 via connection 645. Output port[k,m] 660 thenserializes the frame of 64 bits (in the preferred embodiment), convertsthe electrical signal to a serial optical stream, and injects the bitstream onto optical fiber 665 leading to HBA j. The details of thisconversion and transmission process are well known to practitioners ofthe art of serial digital data transmission via optical fibers.

Externally (messages flowing into or out of the system of hosts, HBAs,and DDMs) one of the common protocols, such as Ethernet, may be used.Internally, messages are wrapped in the SOM and EOM for efficient andreliable transfer between hosts. In normal operation, the host sends adata packet in the form of destination, priority, message length and asequence of bytes containing the message payload to its HBA. The payloadcontains arbitrary information and may include other communicationprotocols for routing outside the MDS. The function of the HBA is todecompose these data into frames (of 64 bits in the preferredembodiment), prefix the sequence of frames with the SOM control framewhich contains information as to the packet's destination, priority, andoffset into the receiving host's memory for the RDMA function. The HBAalso prepares and appends the EOM control frame which contains a 32-bitCRC error-detecting code and a source designation identifying thetransmitting HBA.

FIG. 7 illustrates the MDS' transmission and control protocol in thepreferred embodiment. Variations on this basic theme in keeping with theessential functions of group identification for multicast, DMA accessinto the destination memory, and reliability are, of course, possible.Each of these control frames is 64 bits long with the first section ofthe frame reserved for the control type (SOM 700 in the case of SOMheader, EOM 720 in the case of EOM tail, FCM 740 for FCMs, and CTRL 760for other types of control frames). These frame-type identifiers (700,720, 740, and 760) are decoded in Input Box 310 and result in differentactions in the input box 310 as discussed above. Each control frame haserror code 799 occupying the final byte of the frame. This code may beused for error detection or error correction, depending on the algorithmused to compute the code; both operations are possible, but generallyrequire different state machines in the HBAs and a different error-checkin during the input-stage process.

The control-type identifier takes from 6 to 12 bits depending on theserial interface used to transmit and receive the information. In thepreferred embodiment, these identifiers require 4 bits allowing for 16different control types. The prefix bits 66:64 in the 64b66b encodingidentify any frame as to a control or a data frame (or an error)according to the standard protocol for this encoding. Dest field 702contains a group identifier of 12 bits in the preferred embodiment,allowing a full 2048 groups to be identified in the DDM (whosesubscription table, in this case, would contain 2048 entries). Offsetfield 704 contains an offset into the memory of the destination offsetfrom a base address, which specifies the start of the memory reserved tothe specific group receiving the message.

The EOM is constructed in the same manner with EOM 720 including of 4bits in the preferred embodiment. Field port ID 722 contains a numericalidentification of the HBA sending the message. This identification maybe used in the receiving HBA or host to prepare an acknowledgement (ACK)or negative acknowledgement (NAK) to demand a resend should the MDSoperate under a specific error-recovery protocol. The EOM frame alsooptionally contains priority 724 field that may be used for a variety ofsecurity and recovery purposes (not discussed in this disclosure). Themain function of the EOM frame is to convey the CRC 32 field 726 so thatthe integrity of the data frames lying between the SOM and EOM may beverified. This check is made in the receiving HBA, where, in case of anerror, the entire message may be dropped or reported to the host so thata recovery or retransmission-request process may be initiated.

Each CRC 8 error code 799 contains a check sum or error-correcting codeas describe above to protect the integrity of the first 56 bits of thecontrol frame. In this way, essential control frames, such as FCMs forflow control, may be acknowledged via an ACK control frame (CTRL 760).Error code 799 for the SOM frame causes the frame and entire message tobe dropped in case of an error. This prevents a catastrophic systemerror where the wrong host could have its memory overwritten or thewrong memory location in a valid host could be overwritten. Since errorcode 799 is checked in input processor channel 430, the transmitting HBAmay be notified immediately by an error FPM CTRL 760 passed directly tothe correct output FIFO 630. In this case, the transmitting host wouldhave the option to resend the message. If an error in the SOM isdetected in the receiving HBA, the receiving host would have the optionto request a resend by extracting the identification of the sending hostfrom the EOM.

The FCM control frame contains priority field 742 that contains a bitmap of the combined state of the Partition FIFOs[k,p,j] receivinginformation from HBA j. This allows HBA j to either halt or restart atransmission from any of the Priority FIFOs. The FCM control frame alsoincludes reserved field 744 for information input.

Other types of control frames, such as CTRL 760, are comprised of destfield 762 which contains a group identifier of 12 bits, as previouslydescribed. The control frame also includes special information field 764for information input.

Error code 799 in an EOM transmission to the DDM would initiate much thesame action as described in the previous paragraph. An EOM errordetected in the receiving EOM may require a more general type of errorrecovery. However, if error code 799 were an ECC (error-correcting code)such as Hamming (63,57) code, most all errors in the control framescould be corrected without the need for error-recovery in the host. Ofcourse, this depends on the random bit-error rate experienced by thephysical transport layer (in the preferred embodiment, this rate isexperimentally 10⁻¹⁶, meaning that the frequency of occurrence of anuncorrectable error in a control frame would be vanishingly small).

It is seen, by the above discussion, that the control frames presentedin FIG. 7 allow a versatile and secure operation of the MDS. Dataintegrity is protected end-to-end, from transmitting HBA, through theDDM and over the fiber connections, to the receiving HBA. Multicastmessages (see the discussion on groups below) are supported at both theinternal protocol level and by the processes inherent in the DDM frominput box 310, through middle box 330, to output box 360.

The above detailed description of the drawings explains the relationshipbetween the message header and the self-routing process that is inherentin the SOM's various fields and the mechanism of message managementillustrated in FIGS. 3, 4, 5, and 6. A destination for a messageincludes two parts: the group ID and the offset. The group ID, in thepreferred embodiment, serves as an index into a table of groupsubscriptions, said table being located in input box 310 for efficientaccess to the process that directs any message to the correct partitionsand destinations.

Groups are defined at the application level and their table entries arecommunicated to the DDM by maintenance messages generated by a groupmanager process in a particular host. Such messages are directed the DDMitself and are not passed through to other hosts.

Fast priority messages, such as the FCMs, allow the MDS to respondquickly and efficiently to a variety of situations, from FIFO managementto reporting component failures in a timely fashion. Such messages arequick to transmit since each includes a frame of 64 bits that istransmitted at wire speed to any location in the cluster. The brevityand speed of these control messages enables a rapid system response tochanging conditions as well as supports a graceful degradation of thesystem when hosts or HBAs are lost.

The full impact of the self-routing packet is not fully evident until afabric of multiple interconnected MDS modules are deployed. At each stepacross a fabric, the SOM opens a path in the most direct and efficientmanner available to it at the moment it enters a fabric module. Pathsegment availability information is continually updated throughout thefabric by the back-pressure concept based on the FCMs as discussedabove. The advantage is that a global view of the state of messagetraffic is not required; indeed, a global supervisory control functionbased on such global knowledge, with its attendant reliance onspecialized spanning-tree algorithms, is known to be unreliable andcertainly adds to latency across the fabric. These issues are simply notpresent in a DDM fabric where each SOM effects path decisions as themessage traverses the fabric.

A DDM fabric includes multiple identical DDMs interconnected accordingin a specific topological arrangement such as a Clos or hypercubetopology, to give two examples. An example of the Clos topology is shownin FIG. 8 which includes three layers of DDMs each with 8 bidirectionalports, is labeled by its position in the two-dimensional array with thefirst index referring to the row and the second to the column positionin the array. Connections 800 and 860 are to the HBAs in the bottom andtop rows, respectively, while connections 820 is between row 1 and row 2HBAs and connections 840 are between row 2 and row 3 HBAs such that eachHBA in a row is connected to every HBA in an adjacent row. Row 1 and row3, labeled 810 and 850, respectively, comprise the outer rows while row2, labeled 830 is the inner row. A Clos fabric based on modules with nports contains 3/2 n fabric modules (interconnects), 2(nl2)2 connectionsto HBAs, and n²/2 interior connections between rows. The maximum numberof hops to reach any HBA from any other, also known as the fabricdiameter, is 3 for the Clos network. This topology is significantly moreefficient in number of hops and hardware required than a tree structureinterconnecting the same number of HBAs.

For such a fabric to support the inherent advantages of the self-routingmessages described above, the subscription tables for multicast routingmust be tailored to the specific topology. In the case of the Closnetwork of FIG. 8, each DDM requires a subscription table based on itslocation (or index pair) within the fabric so that the destinationcarried by an entering SOM is mapped to the correct output ports. Thedetails depend on the assignment of HBA indices as well. The computationof these tables is based on a simple algorithm and the tables may beupdated, with group membership changing as the application demands. Eachsubscription table is maintained by the process described above formaintenance messages.

Embodiments of this disclosure can include the use of a simplesubscription table containing an exit map for each defined group. Inembodiments of this disclosure, traffic flow through a fabric ofidentical DDMs depends the contents of the subscription table in eachDDM.

Embodiments of this disclosure can include the fast-priority messagethat uses the same data links between MDS components as other messages.Most interconnect systems are based on Ethernet or Infiniband that bothrequire much longer control messages than the fast-priority message 64bits and/or are made over separate physical connections. Agile flowcontrol based on the fast-priority message does not require additionalconnections or access to a control plane or supervisory traffic manager.

DEFINITIONS

The phrase end-to-end partitioning of message pathways is intended tomean partitioning of the message pathways from a CSME (computing,storage, or network element) to another CSME, for instance a prioritychannel from a computing element through a host-bus adapter through adata distribution module through another data distribution module thenthrough another host-bus adapter and then to a storage element. Thephrase multiple priority levels is intended to mean three or morepriority levels, for instance five priority levels including a highestpriority channel reserved specifically for fast priority messages and achannel reserved specifically for maintenance functions. The termsprogram and software and/or the phrases program elements, computerprogram and computer software are intended to mean a sequence ofinstructions designed for execution on a computer system (e.g., aprogram and/or computer program, may include a subroutine, a function, aprocedure, an object method, an object implementation, an executableapplication, an applet, a servlet, a source code, an object code, ashared library/dynamic load library and/or other sequence ofinstructions designed for execution on a computer or computer system).

The term uniformly is intended to mean unvarying or deviate very littlefrom a given and/or expected value (e.g., within 10% of). The termsubstantially is intended to mean largely but not necessarily whollythat which is specified. The term approximately is intended to mean atleast close to a given value (e.g., within 10% of). The term generallyis intended to mean at least approaching a given state. The term coupledis intended to mean connected, although not necessarily directly, andnot necessarily mechanically.

The terms first or one, and the phrases at least a first or at leastone, are intended to mean the singular or the plural unless it is clearfrom the intrinsic text of this document that it is meant otherwise. Theterms second or another, and the phrases at least a second or at leastanother, are intended to mean the singular or the plural unless it isclear from the intrinsic text of this document that it is meantotherwise. Unless expressly stated to the contrary in the intrinsic textof this document, the term or is intended to mean an inclusive or andnot an exclusive or. Specifically, a condition A or B is satisfied byany one of the following: A is true (or present) and B is false (or notpresent), A is false (or not present) and B is true (or present), andboth A and B are true (or present). The terms a and/or an are employedfor grammatical style and merely for convenience.

The term plurality is intended to mean two or more than two. The termany is intended to mean all applicable members of a set or at least asubset of all applicable members of the set. The phrase any integerderivable therein is intended to mean an integer between thecorresponding numbers recited in the specification. The phrase any rangederivable therein is intended to mean any range within suchcorresponding numbers. The term means, when followed by the term “for”is intended to mean hardware, firmware and/or software for achieving aresult. The term step, when followed by the term “for” is intended tomean a (sub)method, (sub)process and/or (sub)routine for achieving therecited result. Unless otherwise defined, all technical and scientificterms used herein have the same meaning as commonly understood by one ofordinary skill in the art to which this present disclosure belongs. Incase of conflict, the present specification, including definitions, willcontrol.

The described embodiments and examples are illustrative only and notintended to be limiting. Although embodiments of the present disclosurecan be implemented separately, embodiments of the present disclosure maybe integrated into the system(s) with which they are associated. All theembodiments of the present disclosure disclosed herein can be made andused without undue experimentation in light of the disclosure.Embodiments of the present disclosure are not limited by theoreticalstatements (if any) recited herein. The individual steps of embodimentsof the present disclosure need not be performed in the disclosed manner,or combined in the disclosed sequences, but may be performed in any andall manner and/or combined in any and all sequences. The individualcomponents of embodiments of the present disclosure need not be combinedin the disclosed configurations, but could be combined in any and allconfigurations.

Various substitutions, modifications, additions and/or rearrangements ofthe features of embodiments of the present disclosure may be madewithout deviating from the scope of the underlying inventive concept.All the disclosed elements and features of each disclosed embodiment canbe combined with, or substituted for, the disclosed elements andfeatures of every other disclosed embodiment except where such elementsor features are mutually exclusive. The scope of the underlyinginventive concept as defined by the appended claims and theirequivalents cover all such substitutions, modifications, additionsand/or rearrangements.

It will be appreciated by those skilled in the art that changes could bemade to the embodiments described above without departing from the broadinventive concept. It is understood, therefore, that this disclosure isnot limited to the particular embodiments herein, but it is intended tocover modifications within the spirit and scope of the presentdisclosure as defined by the appended claims.

The invention claimed is:
 1. A method, comprising operating a datadistribution system including a data distribution module and a pluralityof host-bus adapters coupled to the data distribution module including:defining a coherent group that includes a set of members that includesthe plurality of host-bus adapters; providing a mirror group-coherentmemory area in each of the set of members; initiating a one-to-allbroadcast message from a one of the set of members to each member of theset of members, including the one of the set of members, ensuring thatwhen the one of the set of members requests a write to its mirrorgroup-coherent memory area, the one-to-all broadcast message maintainstemporal memory coherency across all members of the set of members ofthe coherent group; and, wherein the mirror group-coherent memory areain each of the set of members is physically mirrored with a temporalcoherence in that reads are not delayed or controlled by semaphores andno semaphores or access enables are required to achieve the temporalcoherence of the coherent group.
 2. The method of claim 1, wherein theone-to-all broadcast message is transmitted on a priority channel. 3.The method of claim 2, wherein the priority channel is lock-free andsemaphore-free.
 4. The method of claim 1, further comprising ensuringthat when the one of the sets of members requests the write theone-to-all broadcast message maintains temporal memory coherency acrossall of the set of members of the coherent group including the one of theset of members that request the write.
 5. The method of claim 1, whereinthe coherent group includes multiple priority levels and wherein themirror group-coherent memory area in each of the set of members isphysically mirrored with a temporal coherence on each multiple prioritylevel and no semaphores or access enables are required to achieve thetemporal coherence of the coherent group.
 6. An apparatus, comprising: adata distribution system including a data distribution module and aplurality of host-bus adapters coupled to the data distribution moduleand a processor coupled to a memory; the memory including a set ofinstructions, that when executed by the processor, causes the apparatusto carry out the steps of: defining a coherent group that includes a setof members that includes the plurality of host-bus adapters; providing amirror group-coherent memory area in each of the set of members;initiating a one-to-all broadcast message from a one of the sets ofmembers to each member of the set of members, including the one of theset of members, ensuring that when the one of the sets of membersrequests a write, to update its mirror group-coherent memory area, theone-to-all broadcast message maintains temporal memory coherency acrossall members of the set of members of the coherent group; and, whereinthe mirror group-coherent memory area in each of the set of members isphysically mirrored with a temporal coherence in that reads are notdelayed or controlled by semaphores and no semaphores or access enablesare required to achieve the temporal coherence of the coherent group. 7.The apparatus of claim 6, further comprising a computing, storage ornetworking element coupled to each of the plurality of host-busadapters.
 8. The apparatus of claim 6, further comprising another datadistribution module coupled to the data distribution module.
 9. Theapparatus of claim 6, wherein the coherent group includes multiplepriority levels and wherein the mirror group-coherent memory area ineach of the set of members is physically mirrored with a temporalcoherence on each multiple priority level and no semaphores or accessenables are required to achieve the temporal coherence of the coherentgroup.